[skip ci]
During investigate the flakiness of our CI, I found the root cause for majority of ray application's flakiness is the readinessProbe.
Today in rag application test:

1. We use kubectl wait --all pods -n ml-$SHORT_SHA-$_BUILD_ID-ray --for=condition=Ready --timeout=1200s to wait for the Ray head to be ready. When ray head is ready, it serves the traffic on address: 127.0.0.1:6379
2. However, the readinessProbe is defined as:
   readinessProbe:
   exec:
   command:
   - bash
   - -c
   - wget -T 2 -q -O- http://localhost:52365/api/local_raylet_healthz | grep
   success && wget -T 2 -q -O- http://localhost:8265/api/gcs_healthz | grep
   success
3. The prober only probes the raylet and gcs, but not the ray service on port 6379

Therefore, in our CI tests, we see that when a job is submitted in ray application test, the connection is broken due to the server is not ready.